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Executive Summary 



This research emerged from the Social Science Research Council’s collaborative partner- 
ship with the Pathways for College Network, with technical assistance in data collection 
provided by the Council for Aid to Education. The project has followed over 2,300 students 
at 24 institutions over time to examine what factors are associated with learning in higher 
education. Learning is assessed along the dimensions of critical thinking, analytical 
reasoning and written communication, as measured by the Collegiate Learning Assessment 
(CLA). We consider factors related to individual development as well as patterns of 
inequality associated with disadvantaged groups of students (including students from 
racial/ethnic minority groups, less advantaged family backgrounds, non-English speaking 
homes, and high schools that are comprised primarily of non-white students). Students 
were initially tested at the beginning of their freshman year (Fall 2005) and then followed 
up at the end of their sophomore year (Spring 2007). In addition to the CLA measures of 
learning, supplementary data was collected from student surveys, college transcripts and 
secondary sources of institutional data to generate a Determinants of College Learning 
longitudinal dataset. The scale and scope of this project offers a unique opportunity to 
explore factors associated with learning in higher education. 



Our analysis has identified a broad set of indi- 
vidual, social and institutional factors associated 
with learning in higher education. Identification 
of factors associated with improvement in CLA 
performance can serve to focus policymaker 
and practitioner attention on student experiences 
and institutional practices that are conducive to 
promoting reasoning and communication skills. 
Specifically, our findings include: 

• Students with stronger high school academic 
preparation, measured by both Advanced 
Placement coursework and grade point average, 
demonstrate higher CLA performance as 
entering freshmen, with the gap between 
students who have not had this prior prepara- 
tion significantly increasing over the first two 
year of college. 

• Measures of college engagement exhibit vary- 
ing relationships to growth in CLA scores: some 
forms of engagement are negatively associated 
with improvement in CLA performance 

(e.g., hours spent studying in groups and hours 
spent in fraternities/sororities); while hours 
spent studying alone are positively associated 
with improvement in CLA performance. 



• Working on campus for moderate amounts 
is positively associated with improvement 
in CLA performance over the first two years 
of college; working on campus more than 

1 5 hours per week or working off campus is 
negatively associated with CLA performance. 

• Student perceptions of high faculty expecta- 
tions are strongly associated with improvement 
in CLA performance. 

• Fields of study in college vary to the extent to 
which they contribute to growth in reasoning 
and communication skills as measured by the 
CLA; students concentrating in math, science, 
social sciences and humanities coursework 
have higher levels of improvement than 
students in education, human services or 
business subject areas. 

• Institutional differences in student learning 
as measured by longitudinal changes in CLA 
performance are great; 29 percent of variation 
in longitudinal growth in CLA performance 
occurs across schools. 




Our research also identifies groups of students 
who enter higher education from disadvantaged 
backgrounds and examines how they fare in 
postsecondary institutions over time in terms of 
improving their reasoning and communication 
skills as measured by the CLA. Findings identify 
four dimensions of group disadvantage that are 
associated with both students’ lower initial perfor- 
mance on the CLA as entering freshmen as well 
as, more troublingly, often lower levels of individual 
improvement on this measure over the first two 
years of college. Specific findings on learning 
trajectories of students from disadvantaged back- 
grounds include: 

• Students whose parents completed only a 
high school education or less start college with 
lower CLA scores and progress less on this 
measure during the first two years of college 
than students whose parents obtained graduate/ 
professional degrees. Students from other 
family backgrounds enter higher education with 
lower CLA scores than those whose parents 
hold graduate/professional degrees but gain 

as much over time. 

• Students from families where English is not the 
primary home language start college with lower 
CLA scores and progress less on this measure 
during the first two years of college than students 
from families with English spoken at the home. 

• Students who attended high schools that were 
predominately non-white (i.e., more than 

70 percent non-white) start college with lower 
CLA scores and progress less on this measure 
during the first two years of college than 
other students. 

• Non-white students, including Asian students, 
start college with lower CLA scores and, with 
the exception of Hispanic students, progress 
less on this measure during the first two years 
of college than white students. 



We examined various factors associated with 
learning in higher education to assess the extent 
to which we could identify individual, social and 
institutional determinants of these gaps in CLA 
performance. In particular our research highlights 
the following factors associated with the gaps in 
longitudinal growth: 

• Including high school preparation and individual- 
level college experiences accounts for much 

of the differential rates of growth in CLA 
performance by parental education (the gap 
is reduced by 40 percent and is no longer 
statistically significant). 

• Institutional differences account for approxi- 
mately one-third of the gaps in longitudinal 
CLA performance between African American 
and white students. 

• The gaps in longitudinal growth in CLA perfor- 
mance persist for students who attended high 
schools with predominately non-white peers 
or were from families where English was not 
the primary language, regardless of inclusion of 
the additional individual, social and institutional 
measures examined. 

Overall, the reported findings have important impli- 
cations for policy, practice and research. In terms 
of policy, the research suggests the need to focus 
future social policy not just on increasing access 
to college and reducing student attrition, but also 
on assuring success in terms of learning for 
students attending higher education institutions. 
This project also has important lessons for 
practitioners: institutions vary tremendously 
on the extent to which students attending them 
demonstrate growth on CLA performance. The 
longitudinal findings identified here suggest 
the need for additional systematic future study 
of student learning in higher education. 



Learning in Higher Education 



Mounting pressures to hold higher education accountable for student outcomes over the 
last several decades have culminated in the Secretary of Education’s Commission on 
the Future of Higher Education’s report, A Test of Leadership. Reminiscent of The Nation 
at Risk critique of K-1 2 in the 1 980s, the Commission placed the responsibility for the 
nation’s competitiveness in the global economy on the doorsteps of educational institutions. 
With respect to learning, the Commission noted that “the quality of student learning at 
U.S. colleges and universities is inadequate, and in some cases, declining” (p. 3). Based 
on sobering statistics from the National Assessment of Adult Literacy, the Commission 
urged both improvement and accountability of learning in higher education. 

To avoid the pitfalls of narrowly focused externally imposed measures of learning, a growing 
number of institutions and other higher education agencies are thinking about ways to 
assess and improve performance in this area. A newly released report by the Association 
of American Colleges and Universities and the Council for Higher Education Accreditation, 
for example, urges all institutions to develop “ambitious, specific, and clearly stated goals 
for student learning” as well as “gather evidence about how well students in various 
programs are achieving learning goals.” With learning at the forefront of current discussions 
in higher education, developing new measures of learning and a better understanding of 
factors associated with improvement in students’ performance on cognitive tasks is crucial 

We contribute to this endeavor by studying factors associated with changes in student 
performance of over 2,300 individuals at 24 four-year institutions across the nation. 

This project is conducted in partnership with the Council for Aid to Education (CAE) and 
builds on their large-scale longitudinal study, the Collegiate Learning Assessment (CLA) 
Longitudinal Project. CAE has assessed students’ skills when they first entered higher 
education in the Fall of 2005 and again at the end of their sophomore year, in the Spring 
of 2007. Learning is assessed through the Collegiate Learning Assessment (CLA), which 
relies on open-ended questions to measure broad ability skills such as critical thinking, 
analytical reasoning and written communication. Students’ average spring 2007 CLA 
scores were 0.18 standard deviations higher than their original Fall 2005 performance — 
indicating moderate student growth over the first two years of college on this measure. 
We extended this endeavor to discern how student experiences and institutional contexts 
are related to the development of cognitive skills as measured by CLA performance 
during the first two years of college. 




We begin this report by examining how a range of different factors, from high school 
preparation to college experiences and postsecondary institutions attended, are related 
to changes in students’ performance on cognitive tasks involving reasoning and commu- 
nication over the first two years of college, as measured by the CLA. Following identification 
of these factors we focus on experiences of disadvantaged groups of students, where 
disadvantage is broadly defined to reflect socioeconomic background, race/ethnicity, 
segre-gation of the high school attended, and language spoken in the home. We would 
expect disadvantaged groups of students to perform less well on the CLA upon entry 
into higher education. However, the crucial question is what happens thereafter — do they 
catch up or fall further behind? And moreover, can we identify a specific set of factors that 
may facilitate learning of disadvantaged groups of students and help narrow the gap in 
CLA performance? 

The figure below presents the conceptual framework used in this study. The layout of the 
report does not follow the figure step-by-step, but instead presents a selected set of results 
focusing on students' high school and college experiences. As learning in higher education 
potentially enhances the capacity for life-long success and intellectual development, 
improving learning outcomes for all students and decreasing inequality in learning across 
different groups of students, may not only benefit colleges and universities today but may 
also facilitate students’ participation in the political and economic world of tomorrow. 

FACTORS ASSOCIATED WITH GROWTH ON THE CLA 



Measures of Disadvantage: 


Race/Ethnicity 
Racially Segregated 
High School 
(70+ % minority) 


Parental Education 
Non-English Home 
Language 


Parental Occupation 


Control Variables: 


2005 CLA Score 


Two Parent Household 


Sibling Number 


Gender 


Geographic Region 


Urbanicity 



High School Academic Preparation: 


GPA 


Number of AP Courses Taken 


College Experiences: 

Hours Spent Hours 


Spent Studying with Peers 


Studying Alone 
Hours Worked On Campus 


Hours Worked Off Campus 


Faculty Expectations 
Hours Spent in 


Field of Study 


a Fraternity/Sorority 




College Fixed Effects 






Overview of the Conceptual Model Employed in the Study 





Factors Associated 
with Growth on the CLA 



There is substantial consensus on the funda- 
mental skills students should acquire in higher 
education, such as the universally accepted claim 
that students should learn how to think critically 
(Bok 2006). Agreement on how to measure and 
improve critical thinking, on the other hand, is 
much more tenuous. In this project, we assess 
student learning through the Collegiate Learning 
Assessment (CLA), developed and administered 
by the Council for Aid to Education (CAE). The 
CLA asks students to write essays in response to 
“real world” scenarios. Due to the open-ended 
nature of the questions, the CLA aims to measure 
three dimensions of learning deemed important 
in higher education: critical thinking, analytical 
reasoning and written communication (Klein et 
al., 2008). 1 For the purposes of this study, we 
are focusing on one component of the CLA: the 
performance task measure, an example of which 
is presented in Appendix A. 2 

Previous literature reveals a range of college 
experiences that may influence students’ cognitive 
growth (for a review see Pascarella and Terenzini 
2005). Several key factors appear particularly 
relevant: social and academic integration, 
institutional climate, employment patterns, and 
field of study. Moreover, although the literature 
on cognitive development during college tends 
to focus less on pre-college experiences, we will 
also examine students’ academic experiences 
during high school, and particularly their academic 
preparation. In the course of the project, we 
have considered how each set of experiences 
may be related to cognitive growth. Specifically, 



we employ a multivariate statistical framework 
to examine factors associated with 2007 CLA 
performance task score while controlling in the 
analysis for prior 2005 performance task score. 

A more detailed explanation of the sample and 
statistical methods is included in the appendices 
of this report. 

HIGH SCHOOL ACADEMIC PREPARATION 

Although high school experiences have only 
been given scant attention in prior research 
on cognitive growth during college, academic 
preparation is one of the most important factors 
shaping student success in higher education 
(e.g., Adelman 1999; 2006). We thus begin by 
examining the relationship between students’ 
high school grades and AP coursework with 
growth on the CLA performance task measure 
during college. Figure 1 shows that students who 
took four AP courses, and particularly those who 
took 5 or more AP courses, have a substantially 
higher growth on the CLA performance task than 
students who took no AP courses. Moreover, 
students with higher high school GPA’s experi- 
enced higher growth than those with lower GPA's. 3 

It is important to note that our results control for 
the 2005 performance task score; academically 
prepared students thus not only perform better at 
the point of entry into higher education, but also 
experience higher rates of growth in their cogni- 
tive skills. While most of the current discussions 
regarding the need to improve high school 
preparation focus on low persistence in higher 
education, our findings present another reason 
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Figure 1 Predicted 
2007 Test Score by 
Number of High 
School AP Courses 

Note: Predictions 
based on the full 
model reported in 
Table 3C. All other 
variables are set at 
the mean. 
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to think about high school preparation. Students 
who perform poorly in high school will not only 
start higher education at a disadvantage, but will 
also gain less from higher education over time. 
Two alternative explanations suggested by this 
association are worth considering. First, one 
possible interpretation for this finding is that 
students who have done well in high school 
are selected on a range of unmeasured social- 
psychological factors (such as educational 
motivation, attachment and commitment) that 
continue to be associated with higher rates of 
learning for these students in higher education 
institutions. Alternatively, academically prepared 
students are potentially able to apply their high 
school skills to better take advantage of experi- 
ences in the college environment in ways that 
improve their rate of learning as measured by the 
CLA. Regardless of these alternative interpreta- 
tions, the gap between more and less successful 
high school students on an assessment of their 
ability to reason and communicate is significantly 
increased in the first two years of college. 

SOCIAL AND ACADEMIC INTEGRATION 

A substantial amount of attention in the previous 
literature on college learning has been dedicated 
to understanding whether various operationaliza- 
tions of Austin’s (1993) concept of involvement 
and Tinto’s (1993) social and academic integration 
are related to cognitive development. General 
measures of academic effort and engagement as 
well as specific factors such as amount studied 
or books read improve students’ critical thinking 
(Carini and Kuh 2003; Kuh et al. 1991; Terenzini 
etal. 1995). Similarly, social integration, measured 
by interactions with peers and faculty, is positively 
related to students’ development in measured 
performance on complex cognitive tasks 
(e.g., Astin 1993; Frost 1991; Kuh 1995; Twale 
and Sanders 1999; Whitt et al. 1999). However, 
not all engagement is positive — some forms 
of engagement, such as participation in Greek 
clubs, may not always lead to higher cognitive 
growth (for a review and critique, see Pike 2000). 



We explored a range of different measures of 
engagement/integration, but in this report present 
only the statistically significant results. It would be 
expected that the amount of time students spend 
studying is positively related to cognitive growth. 
However, we find that this is dependent on the 
context of studying, and specifically whether 
students study alone or with peers. As Figure 2 
illustrates, the relationship between studying 
alone and cognitive growth is positive while that 
of studying with peers and cognitive growth is 
negative. 4 Every hour spent studying is not alike. 
These results are reason for pause, given that 
many institutions today emphasize studying with 
peers. It is possible that studying with peers helps 
students integrate better into the college life and 
thus increase their persistence. While persistence 
is a desirable goal, it is important to think about 
other potential consequences of peer studying, 
such as those for learning. Moreover, it is possible 
that particular forms of studying with peers, such 
as institutional sponsored and coordinated study 
groups, have a positive relationship to learning. 
Unfortunately, we are not able to distinguish 
between different types of peer study groups at 
this time. Nevertheless, these results suggest the 
need for a more careful examination of what study- 
ing with peers entails and whether it benefits or 
harms students in developing their cognitive skills. 

Another form of engagement that does not 
appear beneficial for the development of cognitive 
skills, as measured by the CLA, is hours spent 
in fraternities/sororities: the more hours students 
spend in Greek activities, the lower their cognitive 
growth. 5 It is important to note that this finding, 
as well as that for studying with peers, may be a 
consequence of self-selection. The models control 
for the 2005 performance task scores and a range 
of other student characteristics and experiences. 
Nonetheless it is possible that different types 
of students spend time in fraternities/sororities 
as well as studying with peers, and thus that the 
negative relationship is not a consequences of 
participation in those activities, but of the factors 
associated with selection into them. We are not 
able to eliminate that explanation given our data, 
but only to establish that there is a relationship 
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Figure 3 Predicted 
2007 Test Score 
by Employment 
Measures 

Note: Predictions 
based on the full 
model reported in 
Table 3C. All other 
variables are set 
at the mean. 



between specific forms of student engagement 
and cognitive growth. However, even if these 
relationships are due to self-selection, the find- 
ings suggest the need to think more thoroughly 
about different ways through which all students, 
even those not necessarily inclined toward 
learning, can benefit from higher education and 
develop their cognitive skills. 

COLLEGE EMPLOYMENT 

Another dimension of college experience that is 
often considered a reflection of student engage- 
ment is employment. Working on campus is 
perceived to reflect engagement, while working 
off campus represents a lack of engagement with 
the college community. Employment, of course, 
can be consequential for student outcomes not 
only as a proxy for engagement but also due 
to the time commitment involved — hours spent 
working (and commuting to work) are hours that 
cannot be spent in other ways, including reading 
and studying. Several recent studies (Pascarella 
et al. 1994, 1998) suggested that employment 
during college is not consequential for learning. 
However, this contradicts an extensive body of 
literature showing that there is a negative rela- 
tionship between employment, especially when 
involving long hours, and a range of educational 
outcomes, especially persistence and attainment 
(see reviews in Pascarella and Terenzini 2005; 
Riggert et al. 2006). 



The patterns of results in this study, highlighted in 
Figure 3, mimic those for persistence/attainment: 
hours spent working are related to cognitive 
growth as measured by the CLA, but the relation- 
ship varies by whether students are working on 
vs. off campus. Hours spent working on campus 
have a positive relationship to cognitive growth, 
although at a diminishing rate (the relationship is 
curvilinear). 6 In contrast, hours spent working off 
campus have a negative relationship to cognitive 
growth, although this relationship is slightly curvi- 
linear as well. Thus, as was the case for studying, 
every hour spent working is not the same. The 
context of work also matters — only on-campus 
employment is associated with the development 
of cognitive skills, as assessed by the CLA. 

INSTITUTIONAL CLIMATE 

Learning may not only be influenced by what 
students do in college, but also by the overall 
climate of the institution attended. Previous 
findings are largely inconclusive regarding the 
role of institutional characteristics in facilitating 
student learning. However, institutional climate, 
particularly institutional emphasis on scholarship 
and learning as well as students’ interaction with 
faculty, appears to enhance student learning 
(Astin 1993; Hu and Kuh 2003; Lundberg and 
Schreiner 2004; Terenzini et al. 1994). We asked 
students to rate a range of statements about 
their postsecondary institutions and one of them 
stood out in terms of its association with our 
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Figure 4 Predicted 
2007 Test Score by 
Level of Faculty 
Expectations 

Note: Predictions 
based on the full 
model reported in 
Table 3C. All other 
variables are set at 
the mean. 
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Figure 5 Predicted 
2007 Test Score 
by College Major 

Note: Predictions 
based on the full 
model reported in 
Table 3C. All other 
variables are set at 
the mean. 



measurement of growth in CLA performance: 
faculty expectations. 7 When faculty members have 
high expectations students experience substan- 
tially higher gains in cognitive skills, as measured 
by the CLA. This finding supports the classic 
Wisconsin model of status attainment, which long 
argued for the importance of significant others, 
including teachers, for student outcomes (Sewell, 
Haller and Portes 1969). 

FIELD OF STUDY 

Although American higher education tends to 
provide general as opposed to vocationally 
specific credentials, not all bachelor’s degree 
programs are the same. One prominent dimension 
of differentiation is college major. Students choose 
a particular field of study, and by extension a 
particular curriculum, which may be consequential 
for their cognitive development during college. 
Indeed, several recent studies have suggested 
that field of study is related to cognitive growth 
(e.g., Li, Long, and Simpson 1999; Pike and 
Killian 2001). Our analyses confirm the relevance 
of college major. Students majoring in science 
and math as well as those majoring in social 
sciences and humanities exhibit higher growth 
in cognitive skills, as measured by the CLA, than 
students majoring in business. Students majoring 
in engineering, agriculture, and computer science 
also experience more cognitive growth, although 
of smaller magnitude. 8 

These results raise questions about the specific 
curricula and learning experiences afforded by 
different fields of study. To explore these issues in 
more depth, we are examining data from college 
transcripts and are currently carefully examining 
students’ course-taking patterns. Although not 
available at this time, we hope that these analyses 
will provide insights into the curricular trajectories 
of students across different fields of study. In the 
meantime, the presented results urge a careful 
evaluation of college curricula in specific majors. 
Some fields are more conducive to the develop- 
ment of general cognitive skills measured by the 
CLA, such as critical thinking, analytical reasoning, 
and written communication. 



INSTITUTIONAL CHARACTERISTICS 

Previous studies report mixed results regarding 
the role of institutional characteristics in 
facilitating cognitive growth during college 
(for a review see Pascarella and Terenzini 2005). 
Since we have only 24 institutions in the sample, 
instead of considering specific institutional 
characteristics, we choose to examine the extent 
to which overall institutional contexts are related 
to cognitive growth. 

After controlling for the 2005 performance task 
score and a range of individual characteristics 
and experiences, we found substantial differences 
across institutions in the degree to which they 
facilitate cognitive growth. These differences are 
not simply a reflection of different student popula- 
tions served by specific institutions, as many of 
the relevant individual level characteristics are 
included in the model. While the small number 
of institutions prevents an in-depth analysis of 
these differences, presented results demonstrate 
notable variation in the development of cognitive 
skills across institutions, which warrants a more 
careful study in both large-scale national endeavors 
as well as by individual institutions. 9 

Notably, accounting for institutional variation — 
by including institutional “fixed effects” in the 
model — alters some of the previously observed 
relationships between student experiences and 
cognitive skills. In particular, hours spent studying 
alone and hours spent working on and off campus 
are of substantially lower magnitude and no longer 
statistically significant. Moreover, the role of college 
major, particularly the previously strong relation- 
ship between science and math and cognitive 
growth, is substantially attenuated. Thus what 
students do, such as their study habits, employ- 
ment, and major pursued, matters partly in 
relation to where it is done. These findings provide 
further evidence that the institutional contexts in 
which students are embedded are consequential 
for the development of their cognitive skills. 



Patterns of Social 
Inequality on the CLA 



Not all students enter higher education with the 
same level of critical thinking, analytical reason- 
ing, and written communication skills. Variation 
among students emerges through a complex set 
of personal and contextual factors. The overall 
patterns of academic performance in part reflect 
the stratification of society at large and inequality 
in K-12 experiences. Consequently, we expected 
specific groups of students to enter higher educa- 
tion with lower levels of skills assessed by the 
CLA, including students from less educated families 
and racial/ethnic minority groups, students 
attending segregated high schools that are 
predominately non-white, and students for whom 
English is not the primary home language. 10 The 
crucial question is: What happens after students 
enter higher education? Do disadvantaged 
groups of students learn at similar rates to their 
more advantaged peers, do they catch up, or fall 
even further behind? The answer to that question 
depends on the social axis or form of disadvan- 
tage examined. 

Our conceptualization of social disadvantage 
is grounded in prior sociological research on 



schooling that suggests how different aspects 
of disadvantage might potentially be related to 
learning outcomes. Specifically, James Coleman 
(1966) argued that racial segregation of high 
schools shaped student peer climates and 
had profound effects on shaping educational 
aspirations, expectations, norms and behaviors 
associated with student performance. These high 
schools also typically suffer from fewer resources. 
Theories of variation in performance by race 
include resistance theories advanced by scholars 
such as Ogbu and Fordham (1986) as well as 
theories of stereotype threat advanced by Steele 
and Aronson (1995). Effects of parental education, 
or social background more broadly, have been 
theorized to have long term effects on expecta- 
tions, attitudes and behaviors related to student 
educational trajectories by sociologists such 
as Christopher Jencks (1972) and Robert Mare 
(1980). Effects of home language on student 
performance have been examined by researchers 
such as Kenji Hakuta (1986) and Min Zhou 
(1997), whose explanatory mechanisms vary from 
cognitive dissonance associated with language 
acquisition to variation in cultural expectations 
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and aspirations to the significance of segregated 
peer environments. 

Figure 6 reports descriptive results for the CLA 
performance task scores in the Fall of 2005 and 
Spring of 2007 across different racial/ethnic 
groups. All racial/ethnic minority groups scored 
lower than white students at the point of entry into 
higher education. While all racial/ethnic groups 
perform less well than whites, the low scores of 
African American students are particularly cause 
for concern given their magnitude and are the 
focus of our analysis of racial differences in this 
report. 11 African American students entered higher 
education scoring substantially (almost one 
standard deviation) below white students on the 
CLA performance task. Even more troubling are 
the patterns of growth over time: African American 
students in our sample gained virtually no 
points on the CLA performance task over time. 
Consequently, the gap between African American 
and White students widens during the first two 
years of college. The extent to which social 
psychological factors such as stereotype threat 
and anxiety over test score performance are 
implicated in these patterns is an important area 
for future research. 



Other racial/ethnic minority groups experience 
slightly more growth, although only Hispanic 
students experience the same growth in cognitive 
skills measured by the CLA as white students. 
Thus, the gaps between white students and stu- 
dents from other racial/ethnic groups increased 
over time, except for Hispanic students. 

The patterns of results also vary across other 
dimensions of disadvantage. As Figure 7 indicates, 
performance on the CLA assessment improves 
with parental education — students from more 
educated families score higher on the CLA perfor- 
mance task upon entry into higher education than 
students from less educated families. However, 
these gaps are not as large as those reported 
for different racial/ethnic groups: comparing the 
two extremes, students whose parents have only 
a high school education or less score approxi- 
mately 2/3 of a standard deviation below students 
whose parents hold graduate or professional 
degrees. Moreover, all groups of students appear 
to develop cognitive skills at approximately the 
same rate during their first two years of college. 
Thus, with respect to parental education, there 
is a pattern of persisting inequality: original gaps 
are largely preserved over time, which is a more 
positive pattern of results than that revealed for 
different racial/ethnic groups. 12 
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LEVEL OF HIGH SCHOOL SEGREGATION AND HOME LANGUAGE 



Figure 8 

2005 and 2007 Test 
Scores, by Level of High 
School Segregation and 
Home Language 



Students for whom English is not the primary 
language scored lower on the 2005 CLA perfor- 
mance task and experienced smaller gains by 
the end of their sophomore year than did students 
for whom English is the primary language 
(Figure 8). The same pattern, but much more 
pronounced, was revealed for students who 
attended high schools where non-white students 
were concentrated (high schools composed 
of 70 percent or more racial/ethnic minority 
students). Students attending high schools where 
non-white students were concentrated entered 
higher education scoring approximately one-half 
standard deviation on the CLA performance task 
below their peers who attended high schools that 
were not predominately non-white. And although 
improving their skills over time, the growth of 
students from high schools with high rates of non- 
white student composition was only about one 
half of the growth experienced by students from 
more racially integrated high schools. 



It may be expected that a part of the disadvantage 
associated with attending high schools that are 
predominately non-white is due to the overlap 
between segregation and other dimensions of 
disadvantage, particularly family background 
and race/ethnicity. Indeed, different measures 
of disadvantage examined in this study are 
correlated with each other. To better understand 
unique contributions of each measure to 
change in CLA performance, we simultaneously 
considered all these factors in a multivariate 
statistical framework. 
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Figure 9 

Test Score Gaps in Baseline 
and Disadvantage Models 

Note: Disadvantage regression 
model predicts 2007 score, 
controlling for the 2005 score 
and the specific dimension of 
disadvantage. The baseline model 
also includes other measures 
of disadvantage and a range 
of background characteristics. 
Non-significant differences 
are shaded. 



Focusing on our measures of disadvantage, 
Figure 9 reports a comparison between disad- 
vantage and baseline models. The disadvantage 
model predicts spring 2007 test scores while 
controlling for fall 2005 test scores and a specific 
measure of disadvantage, while the baseline 
model also includes other measures of disadvan- 
tage and a range of background characteristics. 
For three of the four comparisons considered, 
the gap in the baseline model is significantly 
smaller than that reported when one considers 
the relationship between growth in individual test 
scores and a separately considered dimension 
of social disadvantage. After controlling for other 
dimensions of disadvantage, and other relevant 
factors, the gap between white and African 
American students is reduced by 22 percent. 

The reduction of the gaps between students 
from different family backgrounds (measured by 
parental education) and between students who 
do and do not attend segregated high schools is 
even more pronounced: a reduction of 31 percent 
and 57 percent respectively. The gap between 
students who do and do not speak English as 
their primary language is the only measure of 
disadvantage that is not substantively reduced 
by a substantial amount. 



Thus, some of the stark gaps between more and 
less advantaged groups of students revealed 
in descriptive statistics are reduced when one 
controls for other dimensions of disadvantage 
and other relevant background characteristics. 
Flowever, even after these adjustments, students 
from disadvantaged backgrounds lag behind their 
more advantaged peers in the extent to which 
their CLA test score performance improves over 
the first two years of college. Although of differing 
magnitudes, all differences shown in Figure 9 
in the baseline model that controls for a range 
of social background factors are statistically 
significant. The next section explores the extent 
to which the associations between social disad- 
vantage and improvement in CLA performance 
during the first two years of college can be 
accounted for by considering high school and 
college experiences. 



Explaining Inequality 
in CLA Performance 



To what extent may high school and college 
experience help to explain the observed patterns 
of inequality across groups of students in the 
development of cognitive skills in higher education? 
To consider this question, we examined factors 
discussed in the preceding section: high school 
academic preparation, college engagement/ 
involvement, employment patterns, institutional 
climate, and field of study. We considered one 
set of factors at a time, carefully teasing out the 
extent to which each of those may explain some 
of the gaps between more and less advantaged 
groups of students. 



student concentration (the difference between 
the baseline and full models is only 3 points). 
Finally, the gap between African American and 
White students is not appreciably changed by 
considering these additional factors. Thus, while 
college experiences are associated with gains 
in CLA performance, they do not account for 
differences between more and less advantaged 
groups of students. The only exception is parents’ 
educational background — high school and col- 
lege experiences help to explain a good portion 
of the gap between students from more and less 
educated families. 
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Figure 10 

Test Score Gaps in Baseline 
and Full Models 

Note: Baseline regression model 
predicts the 2007 score, controlling 
for the 2005 score and a range 
of background characteristics. Full 
model also includes measures of 
high school academic preparation 
and college experiences. 
Non-significant differences 
are shaded. 



TEST SCORE GAP 



After accounting for high school and college 
experiences, the gap between students whose 
parents are highly educated (graduate/profes- 
sional degrees) and those whose parents had at 
most a high school diploma has decreased by 
approximately 40 percent and is no longer statisti- 
cally significant (Figure 10). Another difference 
rendered statistically non-significant after including 
high school and college experiences is that 
between students for whom English is and is 
not the primary home language. However, the 
decrease in the magnitude of the difference is 
quite small — only 2 points. 13 Similarly, there is 
virtually no change in the magnitude of the coeffi- 
cient for high schools with high rates of non-white 



Different groups of students may not only have 
distinct experiences during high school and 
college, but may also attend specific types of post- 
secondary institutions. As we noted in preceding 
sections, there are notable differences in the 
growth in cognitive skills across institutions. If 
students from less advantaged groups attend 
institutions with lower average growth, this may 
help to account for some of their lower levels of 
test score improvement during the first two years 
of college. Figure 1 1 presents results for a “fixed 
effect model” — an analysis that essentially removes 
institutional differences from the estimates of 
individual student growth in the CLA performance 
task between the freshman and sophomore years. 
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Figure 11 

Test Score Gaps in Baseline 
and Full+Fixed Effects Models 

Note: Baseline regression model 
predicts the 2007 score, controlling 
for the 2005 score and a range 
of background characteristics. 

Full model with fixed effects also 
includes measures of high school 
academic preparation, college 
experiences, and school fixed 
effects. Non-significant differences 
are shaded. 



After controlling for institutions attended, the gap 
between students from different family back- 
grounds (based on parental education) virtually 
disappears — the difference is no longer statistically 
significant and is of a very small magnitude . 14 
Moreover, the gap between African American and 
White students shows some decline. Although the 
difference is still statistically significant, the African 
American — White gap decreases by approximately 
one-third after controlling for institutional contexts. 
Thus, with respect to family educational back- 
ground and African American race, students from 
disadvantaged groups are more likely to attend 
institutions with lower average growth, which 
helps to explain some of their weaker growth on 
the CLA performance task over time . 15 

The pattern of results for students attending high 
schools where non-white students are concentrated 
and from families not speaking English as their 
primary language in the home is in the opposite 
direction. Although the change in the magnitude 
of coefficients is not dramatic, the gap between 
students from these two disadvantaged groups 
and their more advantaged peers slightly increases 
after controlling for institutional differences. 

This change in coefficients implies that students 
attending high schools where non-white students 
are concentrated and from homes not speaking 



English as their primary language are more likely 
to attend institutions with higher average growth 
then would be expected by random dispersion of 
these groups of students over the postsecondary 
institutions included in our study. 

This final figure presents a somewhat sobering 
set of results. Even after controlling for a range of 
high school and college experiences, as well as 
the specific institutions attended, disadvantaged 
groups of students gain less over time on the CLA 
performance task than other students. The only 
exception to this pattern is family’s educational 
background — differences by family background 
are explained by high school and college experi- 
ences and institutions attended. The failure to 
account fully for the gaps in CLA performance 
during the first two years of college between more 
and less advantaged groups of students has 
prompted us to begin an in-depth analysis of 
students’ academic experiences through the 
analysis of their college transcripts. Although 
not currently available, we hope that analyses 
of course-taking patterns will provide further 
insights into the observed gaps in cognitive skill 
development between more and less advantaged 
groups of students. 



Conclusions 
and Implications 



Students from socially disadvantaged groups 
enter higher education with lower levels of rea- 
soning and communication skills measured by 
the CLA than their more advantaged peers. 
Moreover, gaps between more and less advan- 
taged groups of students either remain stable 
or increase over time (a finding akin to the 
K-12 research on test scores). In addition to 
documenting the gaps in learning, we explore 
how different factors may help to explain the 
observed patterns. Our findings to date suggest 
that high school preparation and institutional 
differences in learning are strongly associated 
with gaps in demonstrated learning between 
different groups of students. 

Overall our findings have implications for policy, 
practice and research. In terms of policy, we 
note that while successfully persisting in higher 
education, some students are not demonstrating 
significant improvement in CLA performance over 
the first two years of college. Public and private 
investment in higher education should be focused 
both on increasing the number of students 
attending college as well as ensuring that students 
are developing skills, including reasoning and 
communication, at reasonable rates of growth. 
Knowing what factors are associated with student 
learning as well as gaps between different groups 
of students can help institutions intervene and 
attempt to counter the patterns of disadvantage 
revealed in this study. 



In terms of practice, we note in general that institu- 
tions matter — 29% of variance in growth in CLA 
longitudinal scores occurs across institutions. 
Colleges and universities also vary extensively on 
the degree to which students from disadvantaged 
backgrounds successfully learn in their respective 
institutional settings. Students in particular are 
sensitive to perceptions of high faculty expecta- 
tions. In addition, students with stronger high 
school academic preparation, studying more 
hours alone, not in fraternities and sororities and 
not working long hours (particularly off-campus) 
learn at higher rates. Students enrolled in math, 
science, social science and humanities also 
demonstrate greater rates of growth on the CLA 
than students in other fields of study. 

Our findings in terms of social science research 
examining educational processes are noteworthy 
for several reasons. First, we note that the CLA, 
as a measure of learning in terms of reasoning 
and communicating in higher education, tracks 
remarkably well with sociological factors at the 
individual, social and institutional level. These 
associations are particularly noteworthy given 
that our measure of growth in CLA performance 
was based solely on the first two years of college 
attendance. In terms of a research instrument, 
although our research did not formally test 
the instruments psychometric properties nor its 
construct validity, the CLA indicator appears from 
a sociological perspective quite promising and 




worthy of further research and development. 

In addition, these findings have important 
implications for the literature on learning which 
has rarely considered inequalities in learning, 
particularly across different dimensions of disad- 
vantage (i.e., studying not only students from 
racial/ethnic minority groups, but also students 
from less advantaged family backgrounds, 
non-English speaking homes, and high schools 
that are predominately comprised of non-white 
students). Moreover, this study broadens the 
extensive literature on inequality in higher 
education, which often focuses on access and 
attainment without considering other aspects of 
college experience, such as student learning. 

We note, however, that our research on individual, 
social and institutional factors associated with 
learning in higher education was based solely on 
non-experimental data. Given that students are 
sorted and self-selected into various high school 
and college experiences, academic programs 
as well as higher education institutions, our find- 
ings here are able to do no more than identify 
patterns of existing associations with variation in 
individual longitudinal growth. Future longitudinal 
research, including experimental and quasi- 
experimental approaches, is required in this area 
to further explore both the character and robustness 
of the associations identified in the observational 
research presented here. 
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FOO!NOTES 

1 Other prominent endeavors aiming to assess learning in 
higher education are the National Survey of Student 
Engagement (NSSE) which relies on student self-reports, 
and the National Study of Student Learning (NSSL), which 
uses the Collegiate Assessment of Proficiency, a multiple 
choice assessment tool developed by the AC! Program. 

For comparisons and trade-offs between different assess- 
ment tools, see Chun 2002. 

2 !he performance task component of CLA was administered 
most uniformly across institutions and is thus the focus of 
our analyses. 

3 Figures in this section report predicted 2007 test scores. 

All models on which these figures are based control for the 
2005 test scores; thus, reported differences across categories 
can be interpreted as differences in growth. 

4 !he coefficients for hours spent studying alone and with 
peers are of similar magnitude if added to the baseline 
model. After controlling for high school academic preparation 
and other college experiences, the coefficient for hours spent 
studying alone decreases by approximately one half. 

s We have asked other questions regarding students’ engage- 
ment, such as their participation in student organizations and 
volunteering. However, those variables were not statistically 
significant. Moreover, using factor analysis, we extracted one 
factor combining different measures of engagement, but the 
Cronbach’s alpha was low and the factor was not related to 
growth in CLA scores. 



6 !he magnitude of the coefficient for hours spent working on 
campus is almost twice as large if only employment is added 
to the baseline model (i.e., without all the high school and 
college experiences). !he coefficient decreases notably after 
inclusion of high school academic preparation. 

7 Students were asked to rate the following statement: “Faculty 
members at my institution have high expectations for student 
like me" on a 7-point Likert scale, ranging from strongly 
disagree to strongly agree. Other statements about students 
(e.g., having high expectations or working hard) or faculty 
(e.g., holding students to high standards or being approach- 
able, helpful, and understanding) were not statistically signifi- 
cant. Moreover, using factor analysis, we extracted one factor 
measuring overall institutional climate, but this factor had no 
statistically significant relationship to growth in CLA scores. 

8 !he coefficient for communications falls just below the 
conventional significance level of p<0.05. As can be noted 
from large standard errors, there is a great deal of variation 
within each field of study. In the future, we will conduct more 
nuanced analyses of course-taking patterns using college 
transcripts. 

9 !he overall pattern of results presented in this report holds 
with and without HBCU institutions included in the sample. 

10 One dimension of potential disadvantage not included in 
this study is gender. In recent years, women’s educational 
performance has equaled or surpassed that of men. More- 
over, there are no significant gender differences in the CLA 
performance task measure between women and men. 

11 After controlling for social background and initial 2005 CLA 
performance, Hispanic students did not differ from white stu- 
dents in their predicted Spring 2007 CLA performance. Asian 
students improved at significantly lower rates than white and 
higher rates than African American students 

during the first two years of college; given relatively high 
initial scores of Asian students, we will examine these 
differences more thoroughly in subsequent analysis. 

12 We also asked students to report their parents’ occupations, 
which we coded into six categories following the EPG 
schema. Parental occupation does not have a significant 
relationship to cognitive growth, as assessed by the CLA. 

13 !he coefficient drops below the significance level since it 
was already close to p=0.05 in the baseline; the change in 
the magnitude is negligible. 

14 Although this difference was rendered non-significant in 
the full model, institutional controls further decrease the 
magnitude of the difference, dropping it to the 35 percent 
of the gap in the baseline model. 

15 For African Americans we found the patterns discussed 
robust, regardless of whether HBCUs were included or 
excluded from the analysis. Consideration of other aspects 
of social background, high school and college experiences, 
as well as institutional experiences does not help to account 
for the gap between white and Asian students in predicted 
2007 CLA performance. 



APPENDIX A: 

CUV Instrument Example 



Collegiate Learning Assessment 

RAND Council for Aid to Education 



Sample Performance Task 



You are the assistant to Pat Williams, the president of DynaTech, a company that makes precision electronic instruments and 
navigational equipment. Sally Evans, a member of DynaTech's sales force, recommended that DynaTech buy a small private 
plane (a SwiftAir 235) that she and other members of the sales force could use to visit customers. Pat was about to approve 
the purchase when there was an accident involving a SwiftAir 235. You are provided with the following documentation: 



1: Newspaper articles about the accident 

2: Federal Accident Report on in-flight breakups in single engine planes 
3: Pat's e-mail to you & Sally's e-mail to Pat 
4: Charts on SwiftAir's performance characteristics 
5: Amateur Pilot article comparing SwiftAir 235 to similar planes 
6: Pictures and description of SwiftAir Models 180 and 235 
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Please prepare a memo that addresses several questions, including what data support or refute the claim that the type of 
wing on the SwiftAir 235 leads to more in-flight breakups, what other factors might have contributed to the accident and 
should be taken into account, and your overall recommendation about whether or not DynaTech should purchase the plane. 



Sample Writing Prompts 



The presents an opinion on an issue and asks the students to address the issue from any per- 

spective^) they wish, so long as they provide relevant reasons and examples to explain and support their views on topics 
such as: "Public figures such as actors, politicians, and athletes should expect people to be interested in their private lives. 
When they seek a public role, they should expect that they will lose at least some of their privacy. " 

The requires students to critique an argument by discussing how well reasoned they find it; they 

must do so by considering the soundness of the argument's logic (rather than agree or disagree with the position pre- 
sented). An example prompt is: “The following is from an editorial in the Midvale Observer, a local newspaper. "Ever since 
the I950's, when television sets began to appear in the average home, the rate of crimes committed by teenagers in the coun- 
try of Alta has steadily increased. This increase in teenage crime parallels the increase in violence shown on television. Ac- 
cording to several national studies, even very young children who watch a great number of television shows featuring violent 
scenes display more violent behavior within their home environment than do children who do not watch violent shows. Fur- 
thermore, in a survey conducted by the Observer, over 90 percent of the respondents were parents who indicated that prime- 
time television programs between 7 and 9 p.m. should show less violence. Therefore, in order to lower the rate of teenage 
crime in Alta, television viewers should demand that television programmers reduce the amount of violence shown during 
prime time." 

Scoring of writing prompts is powered by e-rater, an automated scoring technology developed and patented by the Educational Testing Service and licensed to CAE. 




APPENDIX B: 

Data and Methods 

Determinants of College 
Learning (DCL) Dataset 



Presented data analyses are based on the Deter- 
minants of College Learning (DCL) dataset, which 
was developed in partnership with the Council for 
Aid to Education (CAE). The CAE initiated the 
Collegiate Learning Assessment (CLA) Longitudinal 
Project in the Fall of 2005, administering a short 
survey and the CLA instrument to a sample of 
freshmen at four-year institutions. The same 
students were contacted for the sophomore-year 
follow-up in the Spring of 2007. The Social Science 
Research Council (SSRC) joined the project at 
this time, broadening the original CAE question- 
naire to include a range of questions regarding 
students’ family backgrounds, high school 
characteristics, and college experiences. SSRC 
obtained data from CAE for students who signed 
the SSRC consent form permitting the release 
of their information. This sample included 2,362 
students across 24 four-year institutions. The 
analytic sample used in this report includes 2,332 
students who had valid demographic information 
(race/ethnicity and gender) and test scores for 
both survey years (for descriptive statistics of the 
sample, see Table 1 C). 

Institutions participating in this project include 
a range of schools of varying sizes, selectivity, 
and missions. The sample includes liberal arts 
colleges and large research institutions, as 
well as a number of Historically Black Colleges 
and Universities (HSBU’s) and Hispanic Serving 
Institutions (HSI’s). Participating institutions 
have relied on their own sampling and retention 
strategies. Although CAE has provided overall 
advice and guidance, each institution has worked 
independently to recruit and retain students in 
the sample. The overall retention rate from fresh- 
man to sophomore year across the 24 institutions 
included in the DCL dataset was slightly under 
50%, although this varied notably across institu- 
tions and groups of students. 

Joining the ongoing CAE endeavor has facilitated 
a quick start-up of the project and substantially 
reduced research costs. However, this approach 
has also produced a unique sample of institutions 



and students. To illuminate the characteristics of 
the DCL sample, we have conducted two sets 
of comparisons with the Integrated Postsecondary 
Education Data System (IPEDS). First, we com- 
pared student characteristics in DCL and IPEDS 
for the 24 participating institutions (Table IB). 

With respect to demographic characteristics, 
students in the DCL sample are remarkably 
similar to those attending DCL institutions, 
although the DCL sample includes more women. 
The 75th percentile of SAT/ACT for the DCL 
sample is similar to the overall score for partici- 
pating institutions. The 25th percentile, however, 
is slightly lower indicating that the DCL sample 
includes better academically prepared students 
in the bottom quartile. Following, we compared 
institutions in the DCL dataset to the universe 
of four-year colleges and universities in IPEDS. 
The final set of columns in Table 1 B indicates that 
the demographic and selectivity characteristics 
of the DCL institutions are virtually identical to the 
average for all four-year institutions. 

Furthermore, we compared the characteristics 
of our sample to students in U.S. high schools as 
identified by data combined from two sources: 
the Common Core of Data and the Private School 
Universe Survey (Table 2B). Characteristics of 
both students and high schools represented 
in the DCL sample are quite similar to national 
statistics. The DCL sample includes a lower 
proportion of minority students as well as a lower 
proportion of students from racially segregated 
high schools or high schools with a high propor- 
tion of students receiving free/reduced lunch. 

This pattern is exactly what would be expected 
given the inequality in entry into higher education, 
and particularly four-year institutions. Thus, 
although the DCL sample was not obtained 
through random sampling procedures, students 
and institutions in this sample display a relatively 
close resemblance to students attending 
postsecondary institutions in the U.S. 




TABLE 1 B. 



DCL AND IPEDS DESCRIPTIVE STATISTICS BY GENDER, RACE/ETHNICITY, AND TEST SCORE 







DCL Sample 




IPEDS: 

DCL Schools ONLY 


IPEDS: 

All Schools 


All Schools 


Analysis 

Sample 

Mean 


Freshman 

Mean 


Spring 2007 
Mean 


Freshman 3 

Mean 


All Under- 
graduates 
Mean 


Freshman 3 

Mean 


All Under- 
graduates 
Mean 


STUDENT CHARACTERISTICS 

Male 0.37 


0.40 


0.37 


0.46 


0.46 


0.45 


0.43 


White 


0.59 


0.61 


0.61 


0.61 


0.61 


0.59 


0.60 


African 


0.19 


0.20 


0.20 


0.14 


0.14 


0.13 


0.12 


American 
















Hispanic 


0.05 


0.06 


0.07 


0.08 


0.07 


0.13 


0.12 


Asian 


0.11 


0.10 


0.14 


0.10 


0.09 


0.06 


0.06 


Oth er b 


0.05 


0.06 


0.06 


0.07 


0.09 


0.09 


0.09 


TEST SCORES 
















SAT, 


1052.83 


N/A 


N/A 


995.15 


N/A 


993.14 


N/A 


25th percentile 
















SAT, 


1212.83 


N/A 


N/A 


1219.02 


N/A 


1219.23 


N/A 


75th percentile 
















ACT 


22.05 


N/A 


N/A 


20.86 


N/A 


20.33 


N/A 


25th percentile 
















ACT 


26.29 


N/A 


N/A 


25.77 


N/A 


25.31 


N/A 


75th percentile 

















Notes: IPEDS schools weighted by size. 

“First time, degree seeking undergraduates. 

“For the IPEDS sample, this includes American Indians, students of unknown backgrounds, and non-resident aliens. 
For the DCL sample, this includes American Indians and any students who self-identified as “other” race/ethnicity. 





TABLE 2B. 



HIGH SCHOOL DESCRIPTIVE STATISTICS FOR DCL SAMPLE COMPARED TO ALL HIGH SCHOOLS 



Variable 


DCL 

Sample 

Mean 


All High 
Schools 
Mean 


STUDENT CHARACTERISTICS 


White 


0.69 


0.63 


African American 


0.14 


0.15 


Hispanic 


0.09 


0.16 


Asian 


0.06 


0.05 


American Indian 


0.01 


0.01 


Total Minority 


0.30 


0.37 


SCHOOL CHARACTERISTICS 


Racially segregated high school 
(70% or more racial minority) 


0.13 


0.21 


Charter School® 


0.01 


0.03 


Magnet School® 


0.09 


0.07 


Urban 


0.31 


0.31 


Rural 


0.20 


0.23 


Suburban 


0.49 


0.47 


Northeast 


0.18 


0.18 


Midwest 


0.36 


0.22 


South 


0.26 


0.36 


West 


0.19 


0.24 


Free/Reduced Lunch® 


0.23 


0.32 


Student-Teacher Ratio 


17.30 


18.55 


School Size 


1364.00 


1382.77 



Notes: “Only available for public high schools. 

National data (based on the Common Core of Data 

and the Private School Universe Survey) weighted by school size. 
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COLLEGIATE LEARNING ASSESSMENT (CLA) 

We used the Collegiate Learning Assessment 
(CLA) to assess student learning. The CLA 
measures student learning by asking students 
to write open-ended arguments in response to 
“real world” scenarios. Using these open-ended 
questions, the CLA aims to measure broad ability 
skills such as analytical reasoning, critical thinking, 
problem solving, and decision-making (Klein 
et al. , 2008). Measures used to assess student 
learning consist of three sets of open-ended 
prompts which have been carefully constructed in 
consultation with experts on student assessment 
and tested in a pilot study. The three components 
include: performance task, make an argument, 
and break an argument (see Appendix A for 
examples). Analysis of the pilot CLA study indicates 
that these measures are highly reliable and can 
be obtained relatively economically and within 
a reasonable time frame. Moreover, preliminary 
results suggest that substantial learning occurs in 
higher education, and that the skills acquired by 
students vary across institutions. For more details 
on the CLA study design and measures as well as 
preliminary results, see reports by Klein and his 
colleagues (2003; 2005). 

MEASURES OF DISADVANTAGE 

After examining the overall patterns of learning 
in higher education, we investigated the develop- 
ment of critical thinking, analytical reasoning, 
and expository skills for different groups of students. 
We focused on what are typically considered 
measures of disadvantage in order to understand 
the extent to which broader patterns of stratification 
are manifested in learning outcomes in higher 
education. We began by considering race/ethnicity, 
by dividing students into the following categories: 
white, African American, Hispanic, Asian, and 
other racial/ethnic groups. Following, we consid- 
ered family background, including both parental 
education and occupation. Since parental occu- 
pation did not show a significant relationship to 
growth net of education, we focus the discussion 
on parental education. This variable captures the 
highest level of education completed by either 



parent, grouped in the following categories: high 
school or less, some college, bachelor’s degree, 
graduate or professional degrees. 

In addition to these often discussed measures 
of disadvantage, we examined two other char- 
acteristics: whether a student attended a high 
school with predominately non-white students 
(high school composed of 70% or more racial 
minority students); and whether a student’s 
home language is English (based on the follow- 
ing question: Was English the primary language 
spoken in your home when you were growing 
up?). For each measure discussed, we first 
examined descriptive statistics, considering 
the distribution of 2005 CLA scores, 2007 CLA 
scores, and growth in CLA scores across different 
groups of students (see Table 2C). Following this, 
we examined the relationship between these 
measures of disadvantage and growth in CLA 
scores in a multivariate framework. 

REGRESSION ANALYSES 

Multivariate regression analyses predict the 2007 
performance task score, while controlling for the 
2005 performance task score. Presented results 
thus in effect estimate the relationship between 
specific variables of interest and change in test 
score performance between freshman and 
sophomore years. We focus on the performance 
task since that component of the CLA was ad- 
ministered most uniformly across institutions and 
has the largest completion rates. All analyses are 
adjusted for clustering at the institutional level. We 
used mean substitution for missing data, replacing 
missing values and including dummy variables 
(coded 1) when the substitution was made. 

We began by estimating a baseline model, which 
includes our measures of disadvantage and a 
range of individual-level control variables (gender, 
two-parent household, number of siblings, and 
geographic location of students’ high schools — 
urban/rural/suburban and region). We then 
examined specific high school and college 
experiences: high school academic preparation, 



college engagement/involvement, employment, 
college institutional climate, and college major. 
Our analyses proceeded step by step, consider- 
ing whether and how each set of experiences 
is related to students' performance on the CLA. 
The full model, which is reported herein, includes 
all of the variables of interest. Finally, we also 
conducted a fixed effect analysis by including 
a dummy variable for each of the institutions 
but one in the full model (the institution with no 
growth in CLA scores between 2005 and 2007 
was chosen as a reference). Table 3C reports 
the results of these regression analyses. 

ROBUSTNESS CHECKS: 

CEILING AND MOTIVATION EFFECTS 

We checked our results to ensure that they were 
not an artifact of ceiling effects of the CLA instru- 
ment. We first divided the sample into quartiles 
by high school GPA and SAT. Students in the top 
quartiles on both measures made equal or larger 
gains than students in lower quartiles. Following, 
we conducted a more fine-grained analysis by 
dividing students into deciles based on their 
SAT scores. There is much variation in growth 
across deciles, but no clear indication that higher 
deciles gained less (indeed, students in the highest 
SAT decile showed the largest gains in the 
performance task score between the freshman 
and sophomore years). The same pattern is 
revealed if students are divided into deciles based 
on the 2005 performance task score. While there 
is much variation, students in higher deciles gain 
as much if not more than those in lower deciles, 
and students in the highest decile gain the most. 
Finally, we examined mean growth in CLA scores 
with and without the 10th decile. These analyses 
again confirm that students at the top of the SAT 
or 2005 CLA score distribution are experiencing 
growth over time. 



Moreover, it is possible that reported test scores 
do not only reflect students’ critical thinking, 
analytical reasoning, and written communication 
skills, but also a degree of motivation and effort 
invested in the test. This is a particularly important 
issue to consider given our focus on disadvantaged 
groups of students who may have underperformed 
not because of lower skills but because of lower 
levels of investment. After the completion of the 
CLA instrument, CAE asked students a range of 
questions regarding their experience of the test. 
Among these were questions about effort (e.g., 
whether students engaged in good effort throughout 
the test and gave it full attention) and importance 
(e.g., the extent to which doing well on the test 
was important to students). Based on these ques- 
tions, we used factor analysis to create two scales, 
one measuring the degree of effort invested in 
the test and the other capturing the importance of 
performing well. 

As would be expected, adding these scales to 
the full model indicates that students scored 
higher when they exerted more effort and cared 
more about performing well. However, what is 
crucial for our analysis is that there were only 
relatively small differences in reported effort and 
importance of the test between different groups 
of students (the respective standard deviations 
on these measures are 2.5 and 1 .9; see Table 3B 
for means). Consequently, the addition of these 
scales to the full model did not alter any of the 
results for our measures of disadvantage nor did 
it help to explain any of the gaps between more 
and less advantaged groups of students. Thus, 
while effort and importance of performing well 
matter, they are equally distributed across groups 
and do not explain the observed gaps in the CLA 
performance across our identified groups of dis- 
advantaged students. For brevity of presentation, 
this report includes only the baseline and final 
models, combining all of our variables of interest 
(see Table 3C). Estimates adjust for clustering at 
the institutional level. 



TABLE 3B. 



EFFORT AND IMPORTANCE SCALES BY MEASURES OF DISADVANTAGE 



Variable 


Effort 

Mean 


Importance 

Mean 


RACE/ETHNICITY 






White 


10.45 


6.62 


African American 


10.55 


7.03 


Hispanic 


10.93 


6.89 


Asian 


10.39 


6.71 


Other 


10.59 


6.91 


PARENTAL EDUCATION 






Less than high school 
or high school diploma 


10.53 


6.93 


Some college 


10.65 


6.68 


Bachelor’s degree 


10.48 


6.78 


Graduate or 
professional degree 


10.40 


6.61 


HIGH SCHOOL SEGREGATION 






High school not 
racially segregated 
(less than 70% racial minority) 


10.90 


6.90 


Racially segregated high school 
(70% or more racial minority) 


10.44 


6.69 


PRIMARY HOME 
LANGUAGE STATUS 






English home language 


10.48 


6.71 


Non-English home language 


10.61 


6.80 




APPENDIX C: 
Tables 

TABLE 1C. 

DESCRIPTIVE STATISTICS (N = 2,322) 





Variable 


Mean 


S.D. 


MEASURES OF 
DISADVANTAGE 


RACE/ETHNICITY 

White 


0.64 


0.48 




African American 


0.15 


0.36 




Hispanic 


0.05 


0.23 




Asian 


0.10 


0.31 




Other 


0.05 


0.22 




PARENTAL EDUCATION 

Less than high school or high school diploma 


0.14 


0.35 




Some college 


0.21 


0.41 




Bachelor’s degree 


0.29 


0.45 




Graduate or professional degree 


0.35 


0.48 




PARENTAL CLASS 

Professional 


0.58 


0.49 




Routine non manual 


0.11 


0.32 




Petty Bourgeoisie 


0.08 


0.27 




Laborer 


0.14 


0.34 




Service 


0.09 


0.29 




Racially segregated high school 


0.14 


0.35 




(70% or more racial minority) 
Non-English home language 


0.13 


0.33 


CONTROL VARIABLES 


INDIVIDUAL 

Male 


0.37 


0.48 




Lived in two parent household at 16 


0.83 


0.38 




Number of siblings 


1.73 


1.06 




HIGH SCHOOL CHARACTERISTICS 

Urban 


0.33 


0.47 




Rural 


0.19 


0.39 




Midwest 


0.35 


0.48 




South 


0.30 


0.46 




West 


0.19 


0.40 


COLLEGE EXPERIENCES 


GPA 


3.56 


0.51 




Took 0 AP courses 


0.27 


0.44 




Took 1 AP course 


0.16 


0.37 




Took 2 AP courses 


0.17 


0.37 




Took 3 AP courses 


0.12 


0.33 




Took 4 AP courses 


0.09 


0.29 




Took 5 or more AP courses 


0.18 


0.39 




ENGAGEMENT/INVOLVEMENT 

Hours spent studying alone 


8.72 


6.01 




Hours spent studying with peers 


3.43 


3.39 




Hours spent in a fraternity/sorority 


1.64 


4.57 




EMPLOYMENT 

Hours worked on campus 


4.17 


6.17 




Hours worked off campus 


4.14 


7.36 




INSTITUTIONAL CLIMATE 

Faculty members have high expectations 


5.87 


1.12 




FIELD OF STUDY 

Business 


0.13 


0.34 




Education and human services 


0.07 


0.26 




Engineering, agriculture, and computer science 


0.11 


0.32 




Communications 


0.05 


0.21 




Health services 


0.09 


0.29 




Social sciences and humanities 


0.28 


0.45 




Science and math 


0.16 


0.36 




Undecided, other, and N/A 


0.11 


0.31 


TEST SCORES 


2005 performance task score 


1131.94 


187.76 




2007 performance task score 


1166.26 


21 1 .06 



31 



TABLE 2C. 

CLA PERFORMANCE TASK SCORES BY DIFFERENT MEASURES OF DISADVANTAGE 





2005 


2007 


Difference between 




Performance 


Performance 


2007 and 2005 


Variable 


Task Score 


Task Score 


Performance Task Score 



RACE/ETHNICITY 



White 


1170.39 


1211.47 


41.08 


African American 


994.59 


1001.35 


6.76 


Hispanic 


1102.67 


1151.77 


49.10 


Asian 


1118.67 


1145.68 


27.02 


Other 


1108.05 


1137.23 


29.18 


PARENTAL EDUCATION 

Less than high school or high school diploma 


1063.34 


1102.01 


38.67 


Some college 


1083.18 


1116.50 


33.32 


Bachelor’s degree 


1139.61 


1171.11 


31.50 


Graduate or professional degree 


1183.36 


1218.42 


35.06 


HIGH SCHOOL SEGREGATION 

High school not racially segregated 


1145.82 


1183.51 


37.69 


(less than 70% racial minority) 
Racially segregated high school 


1047.07 


1068.58 


21.51 


(70% or more racial minority) 

PRIMARY HOME LANGUAGE STATUS 

English home language 


1136.36 


1172.30 


35.94 


Non-English home language 


1101.22 


1124.30 


23.09 



TABLE 3C. 

MODELS PREDICTING THE 2007 CLA PERFORMANCE TASK SCORE 



Variable 


Baseline Model 


Full Model 


Full Model with 
Fixed Effects 


MEASURES OF DISADVANTAGE 
RACE/ETHNICITY 

African American 


-105.523*** (19.965) 


-98.437*** (18.387; 


) -70.297*** (16.853) 


Asian 


-31.397* (12.965) 


-36.284** (11.128) 


-34.113** (11.805) 


Hispanic 


-8.400 (22.794) 


-13.033 (19.401) 


-11.504 (20.912) 


Other 


-41.905* (19.724) 


-41.472* (17.795) 


-37.597* (17.427) 


PARENTAL EDUCATION 

Some college 


5.306 (12.128) 


9.202 (11.511) 


11.536 (10.387) 


Bachelor’s degree 


18.293 (13.119) 


11.425 (12.332) 


9.343 (11.181) 


Graduate or professional degree 


42.101** (15.208) 


23.945 (13.680) 


14.910 (11.930) 


PARENTAL CLASS* 

Professional 


7.400 (13.257) 


8.111 (11.915) 


-0.580 (1 1 .699) 


Routine non manual 


10.713 (13.296) 


12.965 (11.704) 


8.174 (10.501) 


Petty Bourgeoisie 


-14.560 (25.629) 


-10.356 (23.128) 


-11.647 (22.782) 


Service 


17.724 (22.950) 


14.758 (22.088) 


11.549 (22.377) 


Racially segregated high school 


-29.186* (14.025) 


-32.901* (12.558) 


-37.299** (13.877) 


(70% or more racial minority) 
Non-English home language 


-28.915* (13.839) 


-26.123 (14.839) 


-37.569** (13.027) 



CONTINUED ON NEXT PAGE 



TABLE 3C. 

MODELS PREDICTING THE 2007 CLA PERFORMANCE TASK SCORE 



Variable 


Baseline Model 


Full Model Full Model with 

Fixed Effects 


CONTROL VARIABLES 
INDIVIDUAL 

2005 test score 
Male 

Lived in two parent household at 16 
Number of siblings 


0.387*** (0.035) 
-0.749 (8.717) 
0.743 (11.261) 
-0.857 (4.713) 


0.294*** (0.027) 
6.092 (9.364) 
-2.148 (10.633) 
2.329 (4.124) 


0.249*** (0.025) 
3.612 (9.309) 
-0.169 (11.194) 
2.945 (4.011 


HIGH SCHOOL CHARACTERISTICS 

Urban 

Rural 

Midwest 

South 

West 


13.726 (11.083) 
-2.076 (11.931) 
17.096 (25.924) 
-34.181 (25.296) 
19.311 (22.841) 


5.795 (10.629) 
-2.359 (12.249) 
12.747 (22.588) 
-33.302 (22.392) 
19.664 (23.808) 


2.722 (9.522) 
-5.443 (12.122) 
7.193 (15.267) 
6.398 (19.001) 
47.967** (17.932) 


HIGH SCHOOL ACADEMIC PREPARATION 

GPA 

Took 1 AP course 
Took 2 AP courses 
Took 3 AP courses 
Took 4 AP courses 
Took 5 or more AP courses 




25.975** (8.288) 
5.602 (10.674) 
5.993 (12.083) 
8.192 (11.780) 
36.171* (14.172) 
62.338*** (11.419) 


29.719** (9.242) 
4.340 (10.676) 
3.525 (1 1 .798) 
8.585 (12.306) 
27.974 (14.704) 
53.241*** (11.005) 


COLLEGE EXPERIENCES 








ENGAGEMENT/INVOLVEMENT 

Hours spent studying alone 
Hours spent studying with peers 
Hours spent in a fraternity/sorority 




2.018** (0.615) 
-4.130** (1.275) 
-3.550*** (0.843) 


0.861 (0.600) 
-4.712** (1.352) 
-2.888*** (0.691) 


EMPLOYMENT 

Hours worked on campus 
Hours worked on campus, squared 
Hours worked off campus 
Hours worked off campus, squared 




4.910* (2.315) 
-0.285* (0.137) 
-4.408* (1.700) 
0.175* (0.084) 


1 .538 (1 .902) 
-0.103 (0.108) 
-1 .967 (1 .305) 
0.085 (0.068) 


INSTITUTIONAL CLIMATE 

Faculty members have high expectations 




1 1 .836* (4.493) 


9.911* (3.747) 


FIELD OF STUDY 15 

Education and human services 
Engineering, agriculture, and computer science 
Communications 
Health services 

Social sciences and humanities 
Science and math 
Undecided, other, and N/A 




0.775 (12.762) 
30.006 (15.378) 
20.776 (15.449) 
24.222 (12.382) 
46.984*** (12.456) 
52.990** (18.430) 
24.441 (16.245) 


-1.854 (12.908) 
23.071 (16.506) 
18.207 (15.084) 
27.996 (10.723) 
32.716** (10.155) 
30.540* (13.584) 
18.151 (14.304) 


SCHOOL FIXED EFFECTS 


NO 


NO 


YES 


Intercept 


727.414*** (44.946) 


630.166*** (60.182) 


677.790*** (43.584) 


R2 


0.270 


0.329 


0.376 



*p<.05, **p<.01, ***p<.001. 

Note: Robust standard errors in parentheses. Analyses are adjusted for clustering of students within schools. 

Missing covariates (with the exception of gender and race) are mean substituted; 

dummy variables flagging missing covariates are included in the analyses but not shown. N = 2322. 

"Laborer is omitted category. 

b Business is the omitted category. 
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